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Annex 1 


Foreword 


The annual meeting of the ENQA IQA Group gathered some 60 participants in the 
premises of the Accreditation Organisation of the Netherlands and Flanders (NVAO) 
in The Hague in June 2009. This seminar was a successful follow-up to the first ENQA 
IQA seminar in Barcelona in 2008 on Internal quality assurance systems and the ESG. 
The seminar in The Hague focused on the theme Internal Quality Assurance - Facing 
Common Challenges. Staff members of ENQA member agencies discussed the common, 
tangible challenges they frequently encounter in their daily operations. The seminar 
drew up a list of the operations that are in paper prescribed as standardised procedures, 
but in practice often turn out to be issues difficult to outline and therefore to measure 
and standardise. Some of the approaches adopted in the seminar included revealing 
some of the key issues in the implementation of procedural improvements; measuring 
of processes and outcomes of an agency’s missions and goals; guaranteeing of efficient 
and excellent operations and following-up on external feedback. 

Discussing common challenges among quality assurance agencies is certainly one of 
the most concrete ways for building a common understanding and thereafter improving 
the internal quality assurance operations of agencies. ENQA fully supports the open 
sharing of challenges in order to gain better ways for the full implementation of the 
Standards and Guidelines for Quality Assurance in the European Higher Education 
Area (ESG) at all levels; in higher education institutions and in quality assurance 
agencies, both externally and internally. 

I do hope that the issues discussed in this publication are of interest and use to 
quality assurance professionals in contributing to the aiming of the European standard 
on accountability procedures and thus to the development of the European Higher 
Education Area. 


Achim Hopbach 
President 

European Association for Quality Assurance in Higher Education (ENQA) 


Chapter 1: 

Assessing QAAs effectiveness 

Douglas Blackstock, QAA, United Kingdom 


1.1 Issue 

This paper contains the three tier scheme for measuring and assessing QAA’s 
performance and effectiveness approved by the Board of Directors of QAA in March 
2008. 

The first report using this scheme was presented to the QAA Board in February 
2009. In light of experience, we will seek to simplify and modify the scheme. 

1.2 Background 

In the course of an extensive discussion of ways to assess strategic performance at its 
meeting in December 2007, the Board concluded that it would be ‘helpful to list out 
relevant sets of quantitative performance measures. . .structured according to the four 
Key Purposes in the Mission statement. This would show that the data we held or could 
collect would help the Board to gauge success in achieving strategic purposes. It would 
also help the Board to identify what other data would be helpful, and what other forms 
of assessment, particularly qualitative assessments, would be needed to make an overall 
judgment about achievement of the Agency’s Purposes. Those qualitative assessments 
might take the form of internal or commissioned research, evaluations and surveys’. 

1.3 Discussion 

The March 2008 proposal aimed to assess effectiveness based on two tiers of 
information - QAA’s performance and QAA’s success - thus informing the Board’s 
assessment of QAA’s impact across the four Key Purposes of QAA. 

The level of operational detail in the first two tiers of information (paragraphs 11 
a and b overleaf) was probably excessive for the Board’s purposes. It was proposed to 
the Board therefore that the Executive should continue to monitor the detail as it does 
now, and that the Board should receive a summary, traffic light report against a limited 
number of headings. That report could be provided quarterly (where the information is 
available quarterly) as part of the Chief Executive’s report to the Board. 

The traffic light summative reporting could be supplemented periodically by 
commissioned research / consultancy or targeted stakeholder surveys. 

At the Board Awayday each October, the Board would review the year’s work and the 
QAA’s achievement of its Key Purposes. 


1.4 Traffic light definitions 

We proposed to adapt the Committee of University Chairmen traffic light definitions as 
follows: 


Good: this is on track; low risk 

Satisfactory: this is broadly on track with some concerns which need to be addressed 

Mixed: some significant concerns which could be detrimental if not addressed; medium risk 

Problematic: serious concerns threaten this area; high risk to the organisation’s overall 
performance 


1.5 The three tiers of information 

The proposed three tiers of information are as follows: 

a. QAA’S PERFORMANCE 

These would be largely quantitative measures, trackable ‘events’ of one sort 
or another, covering performance in the year. Much of this is management 
information that already comes to the Board routinely either through the Chief 
Executive’s reports, other Board papers, the quarterly reports to the funding 
bodies, the annual reports to the Board on strategies (finance, communications, 
HR, etc) or the annual Directors’ Report and Accounts. 

Programmes of work (quarterly monitoring of QAA operating plan) 

• Key contractual commitments met, e.g. reviews and audits completed, reports 
published 

• Progress against non-contractual activities, e.g. papers published, other 
publications, events held, presentations given, attendance at external events 

• Exceptions and achievements 

• Any shift in risk 

External evaluation (some annual, some periodic) 

• External auditors 

• Internal auditors 

• External standards - IS017799 on Information Security, Investors In People, 
W3C web accessibility, Welsh Language Scheme 

• ENQA, Higher Education Regulatory Review Group , Joint Quality Review Group 
(Scotland) 

Staff satisfaction (quarterly and annual) 

• Overall indicators from the annual staff survey 

• Quarterly staff turnover, absence, recruitment analysis 

Financial performance (quarterly and annual) 

• Monitoring and achievement against budget; budget variances 

• Balance sheet 

• Forecasts from quarter 2 onwards 


b. QAA’S SUCCESS 

These would be largely qualitative indicators, and would reflect broadly the 
stakeholder satisfaction with QAA and QAAs work. Some of this is already 
collected routinely, for example evaluations of audits / reviews, demand for our 
‘products’ such as publications, surveys of Higher Quality readers; some of this is 
information that is collected (but not systematically recorded at present) from our 
engagements with funders, representative bodies and governments, for which we 
would need to put appropriate processes in place. 

Effectiveness of processes 

• Institutional and reviewer / auditor evaluations of reviews and audits; reviewer/ 
auditor evaluations of their training; participants’ evaluations of QAA events 

• Renewal of contracts; repeat business 

• New business (from current clients and new clients) 

• Success of ‘engagement’ strategies; students, employers. Professional Statutory and 
Regulatory Bodies 

Adding value 

• Demand for our ‘products’ ; hard copy reports, web traffic, external attendance at 
our events 

• Demand for our services ; advice, contribution, participation, expertise, briefing, 
presentations 

• Hot topics ; our take, our engagement 

• Interactions at government level 

• Interactions with non-HEI stakeholders 

• External review of QAA, such as ENQA membership review 

• External partners, such as QAFRG, HERRG 

Reception 

• The ‘mood’ (in terms of how stakeholders respond to QAA) of the regular 
quarterly meetings with Scottish Funding Gouncil / Universities Scotland, Higher 
Education Funding Gouncil for Wales / Higher Education Wales, Northern Ireland 
visits and the annual meeting with GuildHE / Higher Education Funding Council 
for England / Universities UK 

• The ‘mood’ of the twice-yearly meetings of the Sounding Board group of QAA 
funders 

• The ‘mood’ of, and intelligence from, QAA liaison officers’ interactions with 
individual institutions 

• The ‘mood’ of the Annual Reception and the Subscribers’ meeting 

• The amount, ‘mood’ and tone of press coverage 

• The annual survey of HQ readers 

It was agreed by the Board of QAA that there could periodically be stakeholder 
surveys repeating an exercise carried out in 2006. These could be targeted at 
particular questions that were not answered in the assessments above or could 
be used to ‘test’ the assessments above. The surveying could be done on a larger 
scale than before, for example, by individual HEIs rather than their representative 


bodies, or the 1200 or so organisations on our database. There would also be 
alternative approaches, depending on the audience to be reached, for example, 
by buying questions on MORI or other surveys , or by commissioning research or 
consultancy. 

c. QAA’S IMPACT 

QAA’s four key purposes would represent the top tier of information and analysis. 
The Board would judge to what extent the performance and success data above 
allowed it to make an assessment of the achievement of each Key Purpose. The 
Board would then need to identify whether additional information were needed to 
help it assess QAA’s impact. Ways in which the first two tiers of information might 
inform the third tier are below: 

1. Safeguard the student and wider public interest in the maintenance of 
standards of academic awards and the quality of higher education 

- Audit / review of all institutions according to published timetables and 
methods. 

- Limited and no confidence judgments given where necessary. 

- Publication of reports and judgments. 

- Provision of advice to governments on applications for the grant of degree- 
awarding powers and university title. 

- Institutions with limited or no confidence judgments take the necessary action 
to put things right in the agreed timescale. 

- Periodic summative, evaluative and thematic analyses of reports. 

- Evaluation and review of methods to ensure their continuing currency and 
effectiveness. 

2. Communicate information on academic standards and quality to inform 
student choice and employers’ understanding, and to underpin public policy- 
making 

- Demand for QAA information (hard copy and Web). 

- Demand for our expertise and contribution, including governments and near- 
government. 

- Achievements of our approach to student engagement. 

- Achievements of our approach to employer engagement. 

- HEIs’ use of QAA reports / outcomes as part of their public profile. 

- Potential students’ use of QAA information in making choices. 

3. Enhance the assurance and management of standards and quality in higher 
education 

- Institutions with limited or no confidence judgments take the necessary action 
to put things right in the agreed timescale. 

- Confirmation through the programme of mid-cycle reviews that all institutions 
have reviewed their practice in the light of audit / review recommendations. 

- The Academic Infrastructure is kept up to date and is responsive to the 
changing HE agenda. 


- Development and dissemination of advice and guidance in the light of the 
changing HE agenda. 

- Reduction of regulatory burden, through collaborative work with PSRBs and 
other agencies, whilst maintaining quality and standards. 

4. Promote wider understanding of the nature of standards and quality in 
higher education, including the maintenance of common reference points, 
drawing on UK, other European and international practice. 

- The Academic Infrastructure is kept up to date and is responsive to the 
changing UK and overseas agenda. 

- Demand for QAA advice / contribution to overseas developments, and briefings 
of visitors from overseas. 

- Development and dissemination activities to promote understanding. 

The chart overleaf illustrates the summary position that the Board considered 
at its Eebruary 2009 meeting (based on the 2007-08 financial and operating 
year). Reading from left to right, the Board assessed to what extent each of the 
performance and success datasets inform an assessment of achievement against 
each of the Key Purposes. Having done that, and reading from top to bottom 
down each of the Key Purpose columns, the Board was able to assess to what 
extent each of the Key Purposes was being achieved, and / or what further 
information might be required. 

1.6 Ongoing assessment 

In June 2009 I presented what QAA had learned from this process to the ENQA_IQA 
seminar in the Hague. The presentation can be found at 
http://www.enqa.eu/eventitem.lasso?id=246&cont=pasteventDetail 

So what did we learn from this exercise? 

• The process was very demanding of staff time and we could simplify the process 

• The traffic lights help you focus on managing risk 

• In some areas we needed to improve information management and sharing 

• Some of the information available did not contribute to an assessment of the 
achievement of certain key purposes. 

And that whilst measurement can inform, ultimately, any assessment of the 
achievement of key purposes is a JUDGEMENT based on an array of evidence, not a 
measurement. 

In November 2009 the QAA Board received a further report on ‘QAA effectiveness’ 
based on the financial and operating year of 2008-09. 

In the light of the Board’s discussions in Eebruary 2009 the effectiveness 
framework was simplified to focus on the outward looking, evaluative data 
what we called ‘managing the business’. The internal process data, we called 
this ‘managing the organisation’, much of which is routinely reported to the 
Board through other means (quarterly monitoring. Chief Executive’s reports, 
Einance reports. Annual HR report etc) was captured in a summative set of key 
performance indicators (KPIs). 


The framework was revised as follows: 


2007-08 FRAMEWORK 2008-09 FRAMEWORK 


DATA SETS 

DATA SETS 

QAA’s performance 

* Programmes of work 

• External evaluation 

* Staff satisfaction 

• Financial performance 

Managing the 
business 

• Assurance and 
enhancement 
programmes 

• Effectiveness of 
programmes 

• Relationship with our 
key stakeholders 

• Demand for our services 

QAA’s success 

• Effectiveness of 
processes 

• Adding value 

• Reception 

QAA’s impact 

Traffic light assessment 

QAA’s impact 

Traffic light assessment 



Managing the 

Key performance 



organisation 

indicators (KPIs) 


The set of KPTs that we have developed and now tracked over 2 operating years are 
below. We have selected a mixture of achievement, pattern and trend indicators to 
provide a balanced picture of success in managing the organisation. This is a work 
in progress, we will want to talk to colleagues in other organisations about their 
experiences with KPls, and how we could benchmark QAA’s performance against 
others. 


AREA 

INDICATOR 

Contracted reviews 

% done to schedule 

Publication of reports 

% published within contracted time 

Budget 

% variance from agreed QAA budget^ 

Reserves 

Range allowed in the Board’s reserves policy 

Balance of expenditure 

Proportion of non pay: pay 

Staffing 

Trends (FTE staff numbers) 
Staff turnover (head count) 

Staff absence 

Trends (% of days lost) 

Public information 

Trends (Visitors to our external web site) 
Page views 
Number of visits 

Public enquiries 

Trend 

Complaints — under our published 
complaints scheme 

Trend 


1 Normally the overall budgeted position is breakeven and therefore the variance is as a percentage of budgeted income. 
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PART 1: ASSESSING QAA'S EFFECTIVENESS - SUMMATIVE TRAFFIC LIGHT REPORT 


KEY PURPOSE 


Promote wider 
understanding of the 
nature of standards 
and quality in higher 
education, including 
the maintenance of 
common reference 
points, drawing on 
UK, other European 
and international 
practice 



Reception 


Overall assessment 


Overall assessment 


Overall assessment 


Overall assessment 


KEY 


Excellent: 

this is on track; performing well and is substantially effective. 

Good: this is broadly on track; performance is satisfactory with one or two areas 
which could be addressed. 

Mixed: some significant concerns which could be damaging and result in 
ineffective performance if not addressed. 

Problematic: serious concerns threaten this area; this has the potential to become 
substantially ineffective. 


Chapter 2: 

An outsider’s view on IQA: 

Quality culture or quality procedures 

Lee Harvey, Copenhagen Business School, Denmark 

This paper will explore the extent and nature of the internal quality assurance (IQA) 
within quality assurance agencies (QAA), concentrating on the relationship between 
quality culture and internal quality procedures. It aims to answer two primary 
questions: how to develop a quality culture, and how this is different from having a set 
of procedures? 

One might ask if it is legitimate to ask whether quality agencies as experts on quality 
assurance have a quality culture or not? However, two concerns underpin this paper. 
First, quality procedures are not necessarily indicative of a quality culture. Second, 
despite agencies examining the quality procedures of institutions, there is a paucity of 
evidence, from the outsider’s perspective, about the internal quality assurance of the 
agencies themselves. 

To explore this assertion, three non-European agencies were approached prior to the 
ENQA seminar to see how they perceived internal quality assurance. The first agency 
cited its ISO-9000 certification, specifying that it paid attention to the ‘tricky’ issues 
especially in relation to convincing external stakeholders of an appropriate scrutiny of 
operations. It also referred to its sound documentation system, quality commitment and 
a process of continuous quality improvement. The key indicator, for the agency, was the 
significant increase in external stakeholders’ trust, which was in line with the agency’s 
efforts to improve individual assessor quality and assessment competences. 

The second agency suggested that its internal assurance was manifested through: 
the expertise and behaviour of our auditors; the appointment of qualified staff and 
professional development of staff; the consistency of its audit reports; attention to 
detail; feedback systems on all audits and workshops with evidence of action and 
change; and monitoring of auditee action to check on the effectiveness of the agency. 

In addition, the agency went through an external review every five years similar to the 
audits they undertook of institutions. To this, the agency added that it paid attention to 
its finances and maintained a public relations and communications plan. 

One wonders what the agency would have made of a university claim that its 
procedures were sound if it used a similar set of assertions, citing, for example, 
expertise and behaviour of our teachers; appointment of qualified staff and professional 
development of staff; consistency of curricula; feedback systems on all curricula and 
workshops; monitoring student outcomes to check on the institutions own effectiveness 
and so on. 

The third agency indicated that it was recognised by a national meta-agency. Its 
stakeholders accepted this recognition as appropriate and considered it suitable for 
an accreditation process. The agency explained that it set aside time at its biennial 
Board meeting for development of Board members such as improving consistency in its 
decision-making process. 


Twice in the last seven years, the Board held self-assessment retreats with an 
external consultant. Each member assessed both the Board as a whole and their own 
input into areas of responsibility identified through the self-assessment and consultancy 
process. The self-assessment included: establishing, reviewing and revising the mission 
statement; approving and monitoring the agencies’ programmes and services; strategic 
planning; ensuring adequate financial resources; ensuring sound risk management; 
and enhancing the organisation’s public image. The respondent indicated that the self- 
assessment retreats “have been the best investments we have ever made.” 

The responses from the agencies seemed to be confounding three things: self- 
legitimation and external scrutiny; procedures for their own work; and their own 
internal quality assurance processes. The first day of the ENQA seminar also exhibited 
the same tendencies to confound internal quality processes with external profile. There 
were concerns expressed about the consistency of panels and reports made by visiting 
teams. The efficacy of the peer review process was raised, although its effectiveness 
as a way of undertaking an assessment of quality procedures in institutions remains 
to be evaluated. There were also frequently expressed concerns about the potential 
input or interference from politicians, despite agencies claiming to be independent and 
autonomous. 

Details about internal quality procedures were sketchy, in some cases merely implicit 
because of the small size of the agency. This reflected two tendencies. The first was a 
lack of clear indication of purpose of internal quality procedures, and the second an 
assumption that procedures were equivalent to a quality culture. 

Quality culture is the latest buzzword and the concept is already beginning to lose its 
meaning. The notion of culture is complex (Williams, 1983) as is its extension into the 
idea of quality culture (Harvey and Stensaker, 2008). There are numerous definitions of 
culture, and its meaning has changed over time, not least from an elite notion of ‘high 
culture’ through to a more democratic view of multiple cultures and subcultures. At 
core, though culture is shared, learned and symbolic: culture is away of life. Quality 
culture is also complex and reflects the way in which a group of people (organisation 
through to operational unit) address the issue of quality in their everyday existence. 
There is no right or wrong, good or bad quality culture, although a ‘real’ quality culture 
is ‘invisible’; in as much as it is an integral part of everyday practice rather than a rule- 
bound add-on. 

Quality culture is poorly understood and often implicitly construed as embodying a 
system of internal quality monitoring. However, a set of bureaucratic procedures is not 
the same as a quality culture. Most internal processes do not exhibit the characteristics 
of a lived culture, rather they reflect the rules and expectations of an ‘audit culture’. 
They are fundamentally distrustful and constrained by an externally imposed or 
oriented framework of thinking. In the case of agencies, as suggested above, this is 
dominated by self-perceived need for legitimation. This raises the issue of purpose of 
internal mechanisms, which is fundamental to any notion of quality culture. 

Thus, quality culture is not a set of procedures, much less one that can be imported 
from a different context or imposed on an organisation. There is no simple checklist 
of actions to create a quality culture. However, it is legitimate to ask if there is a way 
to encourage quality culture. In essence, the answer is to require self-reflection on the 
part of the practitioners, something akin to developing a self-critical and reflective 
community of practitioners. 


Some aspects of a quality culture that might be relevant, provided they are 
contextualised, include: ownership, processes and decisions; recognition of the need 
for a system (but not the development of a bureaucracy); a focus on behaviour rather 
than systems; clarity of purpose of any system; an appreciation of different stakeholder 
perspectives; co-operation and working in partnership; community-oriented action 
rather than individualism; inspiring and enabling rather than dictatorial leadership; 
being welcoming of external critical evaluation; embracing self-reflection and taking 
the initiative to improve. 

One possibility, suggested by Harvey and Stensaker (2008), is to develop an ideal- 
type set of quality cultures based on two dimensions, whether individual behaviour is 
group-controlled and whether individual behaviour is prescribed by external rules and 
regulations. A simplistic dichotomisation of these dimensions generates four ’ideal- 
type’ quality cultures (in the Weberian sense), which, for higher education settings, 
are elaborated below (Figure 1). It is important to remember that this is a schema to 
aid reflection and that these are indicative types not idealisations in the common- 
sense meaning of the term. According to Weber (1904), ‘an ideal type is formed by the 
one-sided accentuation of one or more points of view’ according to which ‘concrete 
individual phenomena [. . .] are arranged into a unified analytical construct’; in its 
purely Actional nature, it is a methodological ‘utopia [that] cannot be found empirically 
anywhere in reality’. Further, the two dimensions in the suggested analytic framework 
are but two of a number of potential dimensions and they have been dichotomised for 
simplicity and rather more complex elaborations could have been constructed. 

Figure 1: Ideal-type quality cultures 




DEGREE OF GROUP CONTROL 



STRONG 

WEAK 

Intensity of external 
rules 

STRONG 

Responsive quality culture: led by 
external demands, opportunistic, 
combining accountability and 
improvement, but perhaps also 
sometimes a lack of ownership 
and control 

Reactive quality culture: reward or 
sanction led, task-oriented, doubts 
about the potential of improvement, 
compliance, reluctant (’’beast to be 
fed”) 


WEAK 

Regenerative quality culture: 
internally oriented with strong 
belief in staff and existing 
procedures, widespread, 
experimental, although not 
always adaptive to external 
demands and developments 

Reproductive quality culture: wanting 
to minimise the impact of external 
factors, focusing on sub-units, lack 
of transparency throughout the 
institution, emphasise the expertise 
of the individual 


Adapted from Harvey and Stensaker, 2008 


The four resulting types represent different modes of operating. None are ‘correct’ or 
desirable in general but a particular type may be preferable in a specific context. In 
which case, practitioners should reflect on where they are ‘located’ and decide whether 
they might be better adopting a different cultural type. If so, what might be the 
mechanism to shift to an alternative culture? This requires open self-critique and an 
analysis of future directions. There is no easy solution waiting to be picked off the shelf. 
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Chapter 3: 

External evaluation of HAC: 
Follow-up activities 

Tibor Szanto, HAC, Hungary 


3.1 The Hungarian Accreditation Committee 

The Hungarian Accreditation Committee (HAC) was established by the Hungarian 
Higher Education Act in 1993. It is a national agency with a rather wide scope of 
activities. Accreditation is compulsory in Hungary for programmes and institutions, 
public and private, new and operating alike. Beyond the accreditation decision based on 
minimum criteria, the HAC also includes recommendations for quality improvement 
thus serving the accountability and the enhancement purposes at the same time. 

The HAC has a Body consisting of 29 members (plus currently six non-voting 
members invited from disciplines not represented by regular members) delegated for a 
three-year term, and a Secretariat of currently 20 (17.75 FTE) employees. The HAC has 
19 disciplinary subcommittees who discuss individual external expert evaluations on 
new programmes to be launched and, based on these, make proposals for accreditation 
decisions. For institutional accreditation the HAC appoints external review panels 
(“Visiting Committees”). 

The HAC is a public service organisation funded from the state budget (70-90%) 
and fees paid by HEIs for expert opinion on the accreditation of new programmes to be 
launched (10-30%). 

As to the HAC’s activities, operating programmes were accredited together with 
the given HEI and its faculties in the same procedure in the first round of institutional 
accreditation (1994-2002). In the second cycle however (2004-2010), institutional 
accreditation focuses more and more on the internal QA system of the given HEI, while 
operating programmes are evaluated in a separate, disciplinary accreditation procedure 
where all the same programmes in all HEIs in Hungary are scrutinised at the same 
time. 

Among the other activities of HAC, giving opinion on professorial appointments is 
worth mentioning as being not very usual a task in Europe.^ 

The HAC initiated its first external evaluation in 1999. This review was co-ordinated 
by CRE (now: EUA). The review was financed from a World Bank loan to Hungary, and 
the co-ordinating body was selected according to WB rules (short list). The review team 
consisted of six members, a project manager and five international experts (plus the 
PM), four of them from Europe and one from the US. 

The review encompassed the activities of HAC, and it also covered the Hungarian 
QA system in general with detailed analysis, recommendations for future work and 
system building. The major documents of the review, including the Self-evaluation 


2 See the HAC’s English website at www.mab.hu/english/index.html 


report and the Final evaluation report, were published in hard copy and made available 
on the HAC’s website.^ 


3.2 The second external review of HAC (2007-08) 

Similarly to the first one, the second external review was also initiated by HAC. This 
time, however, the exercise was based on the Standards and Guidelines for Quality 
Assurance in the European Higher Education Area (ESG) and its major purpose was the 
reconfirmation of the HAC’s full ENQA membership. The review was co-ordinated by 
the Ministry of Education and Culture (MoEC) together with the Hungarian Rectors’ 
Conference (HRC).Thus, it was a so-called nationally co-ordinated review. It was 
financed by the MoEC. The review team consisted of three international members, 
including the Chair and the Secretary and three Hungarian members representing the 
Hungarian HEIs, the stakeholders and students, respectively. 

The review was of “type B”, focussing, on the one hand, on the HAC’s compliance 
with the ENQA membership criteria (EMC) and thereby the ESG, and, on the other, 
on the operation of the HAG in general, including the follow-up measures on the first 
external review. The final report together with the TOR and SER were published in 
hard copy and made available on the HAC’s website.'^ 

The major findings of the final evaluation report (June 2008) were as follows: 

• Overall substantial compliance of HAC with EMC/ESG 

• Room for improvement in 4 (5) standards (out of 16) 

• Eollow-up on previous review: 

- most recommendations have been addressed 

- further work needed in several cases 

The final evaluation report included recommendations in five groups: 

1. Eocus of activities (7) 

2. Structure and operations (13) 

3. Role of Minister (3) 


The most important recommendations were the following: 

- more focus on output of programmes 

- move to ex-post institutional evaluations 

- wider view of quality (not just academic content) 

- more focus on improvement (ESG 2.4) 

- decreasing the number of HAC members and subcommittees 

- more stakeholders and international experts in the work of HAC (ESG 3.6) 

- students need voting rights in HAG 

- greater transparency, better communication (ESG 2.3) 

- more system-wide analyses (ESG 2.8) 

- Minister should not decide against HAC opinion 

- state funding of HAC should be more transparent (ESG 3.4) 

- HEIs should pay for all appeals (ESG 3.4) 


4. Internal QA of HEIs 


( 5 ) 

( 3 ) 


5. Einancing the HAG 
Altogether 

Glosely related to ESG 


31 recommendations 
6 recommendations 


3 www.mab.hu/english/doc/extevalhac.pdf 

4 www.mab.hu/english/doc/SelfEval_Teljes%20Kiadvany.doc 


3.3 HAC’s reaction to review results 

The HAC reacted to the review results as follows. 

a. The HAC commented on the final evaluation report (FR) in July 2008. (These 
comments were published together with the SER and the FR.) The major elements 
of the comments were: 

• Certain issues discussed and recommendations made in the FR are beyond 
the power of HAC to act upon. 

• The HAC commented on the interpretation of the review panel of the various 
levels of compliance with the ESC. 

• The HAC expressed its sincere appreciation for the thorough work and 
detailed recommendations made by the Panel. 

b. The HAC distributed the FR in hard copy to all Hungarian HEIs and major 
stakeholder organisations, and made it available also on its website. 

c. The HAC initiated discussions on the FR including the meetings of the 
Hungarian Advisory Board and the International Advisory Board, respectively. 

d. The HAC made and Action Plan (AP) which was approved by the Plenary session 
on 5 December 2008. 

e. The Action Plan was sent to ENQA and put on the HAC’s website.^ 

Upon the 31 recommendations by the Panel, the HAC determined 28 actions in 
the AP. As to the scope of actions, some of them involved proposals to be made for 
legal changes while others aimed at refinement of HAC operations. Responsibilities 
for individual actions were assigned to the HAC President, HAC members, the 
Secretary General, the Strategic Committee and the Quality Development Committee, 
respectively. Since the current term of the HAC expires at the end of 2009, most 
actions were/are to be implemented by this deadline. 

As of June 2009, four actions have been fully implemented, seven proposals have 
been made for amending the Higher Education Act, the implementation of ten actions 
are in progress, and seven actions will be implemented later. 

The four actions having been implemented are the following: 

a. The HAC made a comprehensive analysis of the Bologna transition in 
Hungary from the QA point of view, including proposals for refining the 
system. (Action related to ESG 2.8) A summary of the analysis was presented by 
the HAG President on a national Bologna conference in June 2009. 

b. The chair of each Visiting Committee reports directly to the HAC plenary on 
the given institutional accreditation exercise, presenting the final report himself/ 
herself. (This is just returning back to earlier practice.) 

c. More attention is paid to the accreditation of international joint master 
programmes. The HAC President initiated various meetings with representatives 
of Hungarian HEIs taking part in Erasmus Mundus projects and the issue was 
also discussed at a HAC plenary meeting. 

d. Regular all-staff meetings have been reintroduced at the Secretariat.'’ 

e. The HAC made the following proposals for amending the Higher Education 
Act: 


5 See the document at http://www.mab.hu/english/a_quality.html 

6 Because of the extremely high workload due to the Bologna transition these meetings were held only on a monthly - by-monthly 
basis in the last one or two years. Circular internal e-mails and target group meetings were used extensively instead, beyond the 
intensive daily communication on individual issues. 


• The length of the institutional accreditation cycle should be 5 years instead of 
the current 8 years. 

• The number of HAC members (and also subcommittees) should be decreased. 

• At least one foreign expert should be included in the HAC as a full member. 

• Students should have voting rights in HAC. 

• The Minister should give reasons when deciding against HAC opinion in 
appeals or other procedures. 

• Openings of professorships at HEIs should be advertised internationally. 

• HEIs should pay for all appeals processes. (Currently they do not pay for 
appeals related to giving opinion on professorial appointments.) 

3.4 In conclusion 

Although only one year has passed since the (second) external evaluation of the HAC, 
and hardly more than half a year since the approval of the Action Plan, the results of 
the first follow-up measures are clearly visible. But true, it is only the beginning, and 
there is still a long way to go in order to implement all the remaining actions planned. 

Having been through the second external review, however, the HAC is quite 
confident that the external review of agencies is a very useful tool for serving the 
internal quality assurance of the agencies and thereby the enhancement of the 
operation of the agency reviewed. This, in turn, contributes to the quality enhancement 
of the national higher education system in the given country or region and, moreover, 
effectively helps in creating the mutual trust in national HE and QA systems on the 
European level. 
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Chapter 4: 

Internal quality assurance of agencies: 
Standardisation and contextualisation 

Riitta Pyykkd, FINHEEC, Einland 


4.1 Introduction 

In this paper, I am going to look at the challenges of internal quality assurance of an 
agency from two points of view: as an agency chair, and as an external review panel 
member and chair. 

The European Higher Education Area is built on diversity: every European country 
has national responsibility for higher (as well as other) education, and higher education 
institutions are autonomous, as are the quality assurance agencies. On the other hand, 
there exists mutual recognition of the basic methodology of evaluation since mid 1990s. 
It is based on self-assessment, external assessment, publication of the report, and, most 
often, on a follow-up procedure. 

The roles and functions of quality assurance agencies may differ a lot, e.g. in 
their different status and place in the national QA system, in the division of labor in 
higher education and in QA of higher education, and in the involvement of students 
and external stakeholders in evaluation processes. There is also diversity in resource 
basis, but the agencies are usually financed by stakeholders (governments, ministries, 
consortia of universities, rectors’ councils etc.). The definition of the financial vs. 
operational autonomy is mostly based on national culture and traditions. In principle, 
all agencies agree on the publicity of all results of all evaluations, but in practice, this 
may be limited by national legislation or national culture. All in all, the big question is 
how to balance the national traditions and regulations with the European guidelines. 
One size does not fit all. 

4.2. FINHEEC as a learning expert organisation 

The Finnish Higher Education Evaluation Council (FINHEEC) is an independent 
expert body in connection with the Finnish Ministry of Education. The council is 
appointed by the Ministry from candidates proposed by HEIs and national student 
organisations, and most of the finances come from the Ministry. There is full 
operational independence from the Ministry: FINHEEC prepares its own working plan, 
and the decisions taken by the council cannot be influenced by the Ministry. There 
is, however, a financial dependence. In the Finnish cultural tradition, state money as 
taxpayers’ money is the most neutral resource. Financing, coming e.g. from the Rectors’ 
council or the HEIs, would be considered as giving much more possibilities to partiality. 

There are 12 members in the council, four of which come from research universities, 
four from universities of applied sciences, two from national student organisations, and 
two from working life. All members of the council act as independent experts and do 
not represent their background organisations. The council is usually appointed for a 
four year period, and there is a secretariat of 11 members (situation in 2009). 


The mission and goals are defined by a Government Decree on FINHEEC^, and 
the council and its independence is mentioned in the Universities Act from 2009. 
FINHEEC has the national responsibility for quality assurance of higher education. 
Evaluation of research in the country is conducted by the Academy of Finland. 

The main principle of all operations of FINHEEC since the very beginning in 1996 
has been the enhancement-led evaluation. Higher education institutions have by law the 
main responsibility for the quality of their work, and the task of FINHEEC is to assist 
the HEIs and the Ministry by offering information for the development of education. 
The main types of evaluations today are: audits of the QA systems of HEIs, thematic 
and field-specific evaluations, and evaluation and choice of centres of excellence in 
education. 

The documentation of processes was improved during the years 2008 and 2009. 

The external review panels for FINHEEC evaluations are combined from experienced 
experts, who are trained at a regular basis. FINHEEC has a continuous internal 
feedback and reflection process. Feedback is also gathered after every evaluation, 
both from the target institution and the members of the review panel. All reports are 
published, and a publishing seminar organised after every evaluation. According to the 
surveys, there is generally a good trust between the council and the Finnish HEIs. 

Maintaining the traditionally good relationship and trust between FINHEEC 
and the HEIs also in the future is, however, a big challenge. This is true especially, 
when FINHEEC begins charging HEIs for evaluations that have been free of charge 
for institutions thus far. Another big challenge is how to promote the effect of the 
evaluations within the evaluated institutions, exchange and disseminate the good 
practices identified in evaluations among the HEIs. 

FINHEEC is a rather small organisation, which means that the internal quality 
assurance processes must be systematic, but also integrated and closely connected with 
the daily work. In practice, the main QA mechanisms are daily, informal learning and 
discussion, and continuous internal feedback and reflection. These mechanisms are 
well implemented at FINHEEC. 

4.3 Point of view of an external review panel 

My second point of view will be that of an international external review of an agency. 
Tm looking at this theme both as a member and chair of a review panel, but also as 
a chair of a future target organisation. FINHEEC is a full member of ENQA, but will 
soon undergo a new review, which makes these issues very much present at the council. 

Involvement of international experts provides a substantial benefit, but it also 
emphasises the importance of thorough preparation for a site visit. By preparation I 
mean, firstly, a good knowledge of the national HE system and educational culture, 
which may differ a lot from the reviewer’s own. There can be big differences in the 
operational culture between the agency under review and the background agencies/ 
countries of the panel members. E.g. the degree of openness is highly culturally 
determined, and may differ a lot. The panel members should also have good knowledge 
of the evaluation methods applied by the agency, and, naturally, of the Standards 
and Guidelines for Quality Assurance in the European Higher Education Area (ESG). 

A big - possibly the biggest - question is the relevance of national context for judging 


7 The Government Decree is from 1995 with several amendments; a new Decree will be prepared until the end of 2009. 


compliance. How to combine the reviewers’ previous knowledge and expertise to a 
common understanding of the panel, and how to define the required evidence? 

There are also several organisational questions to solve. The duration of the site 
visit should be long enough for thorough discussions. According to ENQA Guidelines for 
national reviews of ENQA member agencies, the report should be “sufficiently detailed 
to provide satisfactory assurance for the ENQA Board”, but there are different ways 
and modes of writing it. My personal experience, and the experience in FINHEEC, 
is that drafting the report should start already before the site visit, with the self 
evaluation report and other background material. These form a necessary basis for the 
reflections during the site visit. The direction in thinking and writing should always 
be from evidence to analysis, from analysis to conclusions and recommendations. The 
roles of the chair and the secretary are of utmost importance in building common 
understanding in central issues. Involvement of student representatives, especially 
if their participation is not embedded in the national institutional culture, requires 
special attention. 

It is challenging for the panel members to change the standards and guidelines 
into questions and to And the evidence. The willingness and ability to look carefully 
at oneself and to measure own practices against the European expectations, is 
also culturally bound. The review might also have different hidden purposes, e.g. 
strengthening of the national status of an agency. The main hoped-for result should, 
nevertheless, be a development through constructive criticism and discussions. 

International review panels primarily use English as their working language, and as 
the only practical solution for writing the report. Nevertheless, Problems may arise in 
countries where good knowledge of English is not so common. Using interpretation, 
on the other hand, takes time and may cause problems in the terminology. Conducting 
an international review in the national language is ideal, but it is not easy to collect an 
international panel with knowledge of other languages than English, the lingua franca 
of today. 

4.4 Conclusion 

Standards and Guidelines for Quality Assurance in the European Higher Education 
Area (ESG) are intended to be neither too detailed nor too prescriptive. The agencies 
have the right “to reflect in their organisation and processes the experiences 
and expectations of their nation and region”. But, they “must ensure that the 
professionalism, credibility and integrity of the agencies are visible and transparent to 
their stakeholders and must permit comparability to be observable among the agencies 
and allow the necessary European dimension”.® The agencies should fulfill their 
national mandate, follow the national legislation and educational traditions, yet, at the 
same time, confirm with the European standards. At least in the short run, it doesn’t 
seem very realistic to try to effect the national legislation. 

Standards and Guidelines for Quality Assurance in the European Higher Education Area 
(ESG) and ENQA membership criteria are mostly the same, even though they have 
developed as two separate processes. Agencies should follow the ESG, but they have to 
comply with the membership criteria. The real value, now and/or in the future, of listing 
in the register is a question that concerns many of us today, including the agencies, our 
stakeholders and the higher education institutions in particular. 


8 ESG Introduction to Part 3 (2009, 23). 


The two main processes, standardisation and contextualisation of quality assurance, 
are a reality in Europe. At the same time, contextualisation is also a central European 
value, as is the diversity in culture in general. 


Chapter 5: 

Common challenges for QAAs: 

ENQA Review Committee perspective 

Guy Aelterman, NVAO, the Netherlands 


5.1 Starting points 

By definition, quality assurance agencies (QAAs) are tasked with safeguarding 
the quality of higher education (HE) intuitions and/or HE programmes. They are 
responsible for external quality assurance (EQA) in the HE sector. The basis of the 
accreditation/evaluation of an institution or programme is, on the one hand, formed by 
the assessment performed by an independent panel of experts and, on the other, a set 
of criteria used by the team of experts in its assessment. One of the assessment criteria 
deals specifically with internal quality assurance (IQA). 

In their turn, QAAs are subjected to periodical external assessments by independent 
panels of experts that also use a set of criteria as their basis. It goes without saying that 
assessment panels also examine the IQA of QAAs. 

In this presentation, our intention is to examine, from the perspective of the ENQA 
Board (ENQA Review Committee), how independent panels assess the IQA of agencies. 
The IQA of QAAs is, after all, one of the points for attention that is listed both in the 
terms and conditions for ENQA membership and in the Standards and Guidelines for 
Quality Assurance in the European Higher Education Area (ESG) as a condition for full 
ENQA membership and/or for inclusion on the European Quality Assurance Register 
(EQAR). 

The starting point for our discussion is, therefore, that when granting full ENQA 
membership, the ENQA Board (ENQA Review Committee) bases its decision on the 
review report drawn up by the independent panel that assessed the QAA concerned 
with special attention to ENQA membership criterion 7 (accountability procedures) 
and/or ESG criterion 3.8. 

For the sake of clarity, we will set out the two criteria that deal with IQA. 

ESG 3.8 - ACCOUNTABILITY PROCEDURES 

Agencies should have in place procedures for their own accountability. 

These procedures are expected to include the following: 

i. A published policy for the assurance of the quality of the agency itself, made 
available on its website; 

ii. Documentation which demonstrates that: 

• the agency’s processes and results reflect its mission and goals of quality 
assurance; 

• the agency has in place, and enforces, a no-conflict-of-interest mechanism 
in the work of its external experts, Committee/Council/Board and staff 
members; 


• the agency has reliable mechanisms that ensure the quality of any activities 
and material produced by subcontractors, if some or all of the elements in its 
quality assurance procedure are subcontracted to other parties; 

• the agency has in place internal quality assurance procedures which include 
an internal feedback mechanism (i.e. a means to collect feedback from its 
own staff and council/board); an internal reflection mechanism (i.e. means 
to react to internal and external recommendations for improvement); and an 
external feedback mechanism (i.e. means to collect feedback from experts 
and reviewed institutions for future development) in order to inform and 
underpin its own development and improvement. 

iii. A mandatory cyclical external review of the agency’s activities at least once every 
five years. 

ENQA CRITERION 7 - ACCOUNTABILITY PROCEDURES 

There are minor, yet understandable, differences between ENQA criterion 7 and 
ESC 3.8: 

• The guidelines of ESG 3.8 have been integrated into ENQA criterion 7 (part of 
criterion). 

• An extension of requirements concerning the cyclical external review » 

“a mandatory cyclical external review of the agency’s activities at least once every 
five years which includes a report on its conformity with the membership criteria of 
ENQA.” 

5.2 Findings and facts in Review Reports 

We will examine several examples from review reports and the way in which the 
different assessment panels deal with criterion 3.8 (ESG) and criterion 7 (ENQA). 

CASE 1: 

Observations: 

a. Since the last review, the agency has set out to correct a weakness that had been 
pointed out by the experts - the lack of a valid internal quality assurance system. 

b. In order to implement the policy that it had already developed in this area, the 
agency formed a working group in order to create the final version of the system. 

c. The mapping of the system, based on the 11 existing procedures, is available on 
the agency’s website. 

d. Lacking a complete system, the agency has already set up various internal quality 
insurance processes, notably several management loops (consultation, analysis, 
decision-making), in order to collect the viewpoint of: 

• the HE institutions through a questionnaire sent to all of the audited 
programmes at the end of each accreditation campaign; 

• the students, who at the end of each audit visit, write a report on the practice; 

• all of the stakeholders, through working groups and an annual conference; 

• and more recently, the graduates. 

Conclusion: 

The panel of experts concludes that the agency has set up procedures that allow it to 
be accountable for its activities and that, in terms of internal quality assurance, it has 


defined a policy, already has actual procedures, and is applying itself to developing a 

real system - which puts it into full compliance with criterion 7 of the ENQA. 

CASE 2: 

Observations: 

a. The Panel has established that the agency has or is developing clear procedures at 
the Executive and Board levels for planning and monitoring its own performance. 
Key processes include: 

• preparation and monitoring of a Strategic Plan for 2008-2010, supported by 
an annual operating plan; 

• adoption by the agency’s Board of a code of ethics and an internal quality 
system manual for all those providing services or collaborating with the 
agency. 

b. The agency publishes an annual report with details of its activities to ensure that 
its processes and results reflect its mission and goals. 

c. The appointment and recruitment procedures for the agency’s reviewers and 
auditors ensure there are no conflicts of interest. 

d. Recently, for the purpose of this evaluation, the agency surveyed external experts. 
The results constituted sufficient reason for it to make follow-up questionnaires 
mandatory. 

e. A survey of stakeholders was recently undertaken for the purpose of getting 
some feedback about the level of satisfaction of stakeholders across a range of the 
agency’s functions. In an attempt to measure the degree to which stakeholders 
believe the agency is fulfilling the principles of independence, objectivity, 
transparency, efficiency, distance and fairness, questionnaires were sent to 
approximately 11,000 stakeholders. The agency intends to develop and routinely 
implement this external feedback mechanism. 

Conclusion: Eull compliance. 

CASE 3: 

Observations: 

a. The Panel has established that the agency has or is developing clear procedures 
at the Group, Executive and Board levels for planning and monitoring its own 
performance. Key processes include: 

• the preparation and monitoring of an Annual Operating Plan; 

• the adoption by the agency’s Board of a revised framework for measuring 
and assessing the agency’s effectiveness, focusing on the three key areas of 
performance, success and impact. The methodology includes ‘traffic light’ 
indicators; 

• an evaluation policy, underpinning an open and reflective evaluation, 
monitoring and review structure for all agency activities across the sector. 

b. The complex stakeholder context within which the agency operates calls for a 
similarly complex accountability system. The Panel confirms that the agency 
delivers substantial accountability to key stakeholders through a variety of 
mechanisms. 


c. The agency operates a number of internal policies, for example: a data protection 
policy, electronic communications policy, equal opportunities policy, evaluation 
policy and human resources policy. All policies are available online. 

d. The agency publishes an annual report with details of its activities to ensure that 
its processes and results reflect its mission and goals. 

e. The appointment and recruitment procedures for agency reviewers and auditors 
ensure there are no conflicts of interest, and reviewers/auditors may not, for 
example, review or audit their own institutions. 

f. The agency employs a number of mechanisms for internal and external feedback. 
These include: an annual staff survey to monitor the satisfaction of employees, 
monthly staff briefings led by the Chief Executive, a better working scheme to 
encourage staff to suggest ways of improving performance and efficiency. 

g. Regular surveys of external stakeholders’ views. 

It was not evident to the Panel that there had been wide consultation within the 
agency in the drafting of the self-evaluation prepared for this review.” 

Conclusion: Full compliance. 

CASE 4: 

Observations: 

a. The agency has, of its own accord, placed various documents on its website 
(notably the “Internal Quality System” and “Code of Ethics”) and has in place 
an internal quality assurance system which has recently been ISO 9001:2000 
recertified. 

b. The agency also has in place, and enforces, a no-conflict-of-interest mechanism 
in the work of its external experts. It provides a document entitled “Agreement 
regarding confidentiality and adherence with the agency’s ethical principles and 
code of ethics”, which has to be signed by the external experts. All in-house staff 
and experts who participate in the agency’s quality assurance processes must 
abide by the “Code of Ethics”. 

c. However, as the agency states in its self-evaluation report, “feedback on the 
processes carried out by the agency is minimal.” It occasionally employs 
questionnaires and meta-assessments. Thus, the review panel recommends 
building up systematic internal and external feedback mechanisms in order to 
provide information on its own development and improvement and to create a 
common evaluation culture within the entire organisation. 

Given the evidence mentioned above, the review panel concludes that the agency is 

fully compliant with this ENQA criterion. 

CASE 5: 

Observations: 

a. The committee already touched on the issue that experience with the internal 
quality assurance and evaluation system is understandably scarce, given the fact 
that it only became operational this year. Nevertheless, there is a structured and 
differentiated system. Erom the presentation the committee received during the 
visit, the impression is that it is a rather ‘heavy’ and all-embracing system. The 


system should be granted the time to prove itself, but in the process it might be 
sensible to make the core processes a strategic priority. 

b. The agency’s Quality Assurance Protocols (derived from the strategic plan) show 
that the agency is developing an approach to internal quality assurance that could 
help address some of the reported weaknesses. 

c. Measures will be developed to make the interaction between the various internal 
and external factors involved more efficient, more reliable and more transparent. 
The documents suggest that internal quality assurance activities were underway 
last year, but the committee heard little evidence of this. Furthermore the Quality 
Assurance Protocols all appear to have been recently approved, although the 
reference dates may represent recent actions taken on policy documents adopted 
at an earlier time. 

d. Furthermore, the committee feels that board positions for stakeholders might 
very well contribute to greater support and to more active participation in 
the development of the system which may be a serious issue given the risk of 
substantially decreasing benefits in a heavy system of programme accreditation 
based on threshold quality. The committee deems this a potential weakness, and 
therefore cannot conclude that the agency complies fully with this standard. The 
review panel advises the Minister and the agency to include relevant stakeholders 
that are not yet represented, especially students. 

Conclusion: Substantial compliance. 

CASE 6: 

Observations: 

a. There is a considerable degree of accountability in the agency’s work. 
Accountability-oriented procedures include rules for a no-conflict-of-interest 
mechanism for external experts and feedback mechanisms. Also, external reviews 
are carried out frequently. 

b. The role of external stakeholders is, however, limited and the composition of the 
Council is based too much on representation of HE institutions only. 

c. Also, the agency’s own internal quality assurance system is underdeveloped. 

d. Openness to the outside world, in particular when there are critical remarks, still 
has to be developed further. 

Recommendation: Although accountability is dealt with in many respects, a further 

opening-up of the work of the agency is desirable. In this process, clear descriptions and 

the implementation of an internal quality assurance system are essential. 

The Code of Ethics should distinguish between those involved in external evaluation 

and internal evaluation within institutions. 

Conclusion: 

Substantial compliance. 

CASE 7: 

Observations: 

a. Although it has not adopted a published policy for the assurance of its own quality 
according to ENQA regulations, in practice the agency implements a number of 
quality assurance measures. 


b. An internal organisation and process manual of the managing body documents 
the process of accreditation. Open questions or suggested changes to the 
accreditation procedure are addressed in regular meetings of the managing body. 
The managing director collects and disseminates questions and problems relating 
to accreditation centrally. 

c. Follow-up workshops are being staged with the heads of the review teams 
each year in September to give feedback after the completion of the evaluation 
processes. 

d. A series of measures and initiatives are designed to ensure the quality of 
work of the managing body. An organisation and process manual describes 
the organisational set-up of the managing body, as well as the responsibilities 
of the staff according to distinct job profiles. It defines the activities of the 
managing body at the interface between the agency and relevant stakeholders 
and documents all process descriptions involving external quality assurance 
procedures as well as internal administration. 

The evaluators note that the agency has not yet developed a formally differentiated 
internal quality assurance system, but is applying several external feedback 
mechanisms and processes. 

Conclusion: The agency is largely meeting the abovementioned ENQA standards. 

5.3 A reflection on those statements 

Specifically, based on the analysis of the examples given above, the following can be 
stated: 

• Regarding ENQA membership criterion 7 and/or ESG 3.8, the decision of the 
review panels ranges from partial to substantial and full compliance. However, 
with little underpinning of the standpoints adopted or with identical arguments 
for different standpoints; 

• Some panels are going into too much detail, often with quite personal reflections 
rather than with considerations concerning the criterion; 

• IQA is only one of the points for attention in criterion 7/ 3.8; 

• The two points for attention regarding IQA , and specifically mentioned in 
criterion 7/3.8 , can almost always be found in the reports, namely (a) policy 
publicly available (on website) and (b) IQA procedures including internal and 
external feedback. Most of the reports outline these two points for attention, 
albeit in a rudimentary manner in many cases. 

• In many cases, however, review reports devote little attention to IQA. This is 
quite peculiar for organisations dealing with QA. Also, review reports should be 
expected to comprise examples of good practices and comparisons between QAAs. 

5.4 Conclusion 

All things considered we can state that: 

• IQA receives relatively little attention in review reports; 

• Many agencies are undertaking much more with regard to IQA than is shown in 
the reports (or even the SER); 

• Some agencies are failing in their own IQA but this is often not demonstrated, 
detected or argued in the reports; 


• IQA is not only an aspect of criterion 7, but can also be demonstrated in other 
criteria, such as ESG 2.5 Reporting, 2.6 Follow-up procedures and 2.8 System- 
wide analyses. 

Given the diversity of the decisions and the approaches to IQA in the review reports, 
ENQA finds it necessary to compile a book of precedents regarding the standpoints of 
review panels and the ENQA Board with respect to IQA, and to make the information 
provided to expert panels even clearer and more explicit with regard to the significance 
and interpretation of the ESG and ENQA membership criteria, in order to ultimately 
maintain and strengthen the consistency of ENQA decisions. 
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Chapter 6: 

Transparency, accountability and 
stakeholders’ protection in the 
Andalusian Agency for Quality 
Assurance in Higher Education 
and Research: An analysis of our 
reporting and publication policy 

Mariano Lopez-Benitez, Esperanza Vera-Toscano and Elias Eereres, AGAE, Spain 


6.1 Introduction 

The University System of the Autonomous region of Andalusia consists of ten public 
universities with a total student enrolment of 223,511 for the 2007 academic year. 

The governance of this University System was transferred over 20 years ago from the 
National to the Regional Government of Andalusia by a decree (RD 1734/86; June 
13'*', 1986). Soon after, efforts were initiated to develop a quality assurance system in 
the Region. These efforts materialised in the Andalusian University Act (LAU; created 
by the Law 15/2003, of December 22“'*) which expanded on the jurisdiction of the 
government of Andalusia over higher education and research, and established the 
Agency for Quality Assurance in Higher Education and Research of Andalusia 
(AGAE). The 2005 statutes of AGAE (Decree 1/2005, January 11'*' - BOJA 9, January 
M'**, 2005) defined the agency as the main instrument for quality promotion and 
assurance, to be carried out through the assessment, process certification and the 
accreditation of learning and research processes in the Andalusian Knowledge System 
and within the framework of the European Higher Education Area (EHEA). More 
specifically, AGAE’s functions include the assessment of the teaching and research 
activities of non-permanent University staff (for both public and private universities) 
for their long term hiring, as well as the evaluation of teaching performance to 
award incentives. For the latter, the AGAE and the National Agency for Quality 
Assessment and Accreditation of Spain (ANEGA) signed in 2007 an agreement of 
collaboration for the development of the DOGENTIA programme, a programme for 
the evaluation of teaching quality. Teaching performance assessment rises as a key 
aspect as long as educational programmes quality assurance guarantees a suitable 
planning, development and enhancement of the teaching performance. AGAE has 
been a key facilitator, and has assisted universities in preparing their final protocols 
for the teaching staff evaluation. In addition, the agency also plays a central role in 
the evaluation of research proposals, research groups and other research initiatives 
funded by the Regional Ministry of Innovation, Science and Business (GIGE). Lastly, 
AGAE in collaboration with ANEGA is carrying out the evaluation of the new officially 


recognised university degrees and master programmes, following the University degree 
reform undertaken as a result of the Bologna process (Programme VERIFICA). 

These current functions of the agency involve a large number of individuals (both 
AGAE’s employees and many external advisors and experts), and entail a number of 
administrative procedures where decisions have to be made efficiently, that is, promptly 
and based on sound lines of argumentation and reasoning. More important is the 
issue of reporting. As quality assurance agencies we must fulfil our duty in publishing 
not only the process, criteria and procedures used, but also the decisions reached 
within the different programmes in place. This is not an easy task since the privacy of 
individuals and institutions is protected by law, and therefore, we sometimes have to 
struggle with conflicting interests so as to ensure the necessary transparency weighting 
the interests involved. 

Based on AGAE’s experience, this article aims to reflect on this dilemma between 
transparency and the privacy of individuals and institutions. In doing so, we hope to 
bring some light on the question of how to combine and optimise public reporting as an 
Standard for Internal Quality Assurance (IQA) with the assessment and promotion of 
the quality of higher education and research. 

6.2 Overview of AGAE's publication policy 

As mentioned earlier, AGAE was established by the autonomic government of 
Andalusia for the purpose, inter alia, of quality assurance among higher education 
providers. It is a public entity that stems from the Regional Ministry of Innovation, 
Science and Enterprise (GIGE), though with its full legal personality. In consequence, it 
was created as an autonomous organisation, with capacity to manage its own assets and 
to carry out its functions. AGAE is therefore independent of institutional influence in 
its decisions. Nonetheless, as a public agency, it is subject to the Public Administration 
Law, which means that its decisions have to be made public and they must be based 
on sound lines of argumentation and reasoning to the benefit of its stakeholders 
and providers. Meeting all these criteria is, however, not simple. We And that the 
privacy of individuals and organisations is also important, and AGAE guarantees the 
confidentiality of data provided by users and its treatment in accordance with current 
legislation on the Protection of Personal Data. This state of affairs puts this quality 
assurance agency in a situation, where we have to balance between the right amount 
of both publicity and confidentiality, not only to provide the applicant with the correct 
amount of information, but also to avoid engagement in lengthy administrative legal 
proceedings. Here is the debate. 

The AGAE is strongly committed to publishing its reports. As indicated in the 
Agency’s Statutes (art. 7) “the Agency will guarantee the objectivity of the information 
used in the implementation of its functions and provided by any communication 
channel used”; it will also “ensure that the information available contributes to the 
decision making of its stakeholders and providers”. Thus, the agency acts on the basis 
of rigorousness, quality and objectiveness in its processes, governed by the principle 
of transparency with regard to methods, standards and results, in reference to its 
accountability to society (art. 9 of AGAE’s statutes). Given this commitment, AGAE 
presents Annual Reports with aggregate information about its evaluation, certification 
and accreditation activities which are available on its website, and are sent in hardcopy/ 
electronic copy to a wide range of economic and social interest groups (i.e. commerce 


chambers, professional associations, trade unions, business organisations, city halls, 
public administrations, etc.). Local and National newspapers are also informed (see for 
example, El Pais of December, 10'*' 2008 where a summary of AGAE’s 2008 Annual 
Report is presented). 

Other than the Annual Report, the agency provides precise information about all 
its functions to its stakeholders and providers, using the website as the most efficient 
platform for dissemination. Following Law 11/2007, June 22"'*, on citizen’s electronic 
access to public services (arts. 8 and 10-12), it is compulsory for public administrations 
and agencies to implement an electronic device with complete and up-to-date 
information. Thus, the website is AGAE’s main instrument for instant communication. 
All relevant information for the university community and other potential users 
regarding the agency’s services and activities is posted and made available there. That 
is the legal framework of all AGAE’s functions, the guidelines and evaluation criteria 
manuals, the instructions for users, some of our evaluation reports and other news and 
highlights. 

The agency is further working in the elaboration of a “frequently asked question 
(FAQ)” section within each programme. The systematic answer to users’ questions 
comprises a corpus iuris (as indicated in art. 54 of the Spanish Public Administration 
Law) which further increases their security (based on programme’s transparency) 
and shall serve to enhance AGAE’s public trust. In line with this argument, it is 
also important to highlight that not only the evaluation criteria are public but the 
composition of the panels of experts (mostly from outside Andalusia) that advise the 
agency on its different evaluation programmes is also available on the website. As a 
result, and despite the independence and “technical discretionality” of the committees 
in the different evaluation processes which has been protected by several statements of 
our Gourts of Justice, this is limited sometimes given the high level of transparency and 
publicity of AGAE’s processes. 

Notwithstanding, we mentioned earlier that the agency has to provide the necessary 
transparency and publicity (as indicated in art. 105 of our Spanish Gonstitution), 
while ensuring a fair degree of confidentiality when necessary. Regarding the latter, 
the aforementioned Protection of Personal Data Law places great emphasis on the 
protection of the interests of the assessed parties whether individuals or institutions. 
Sometimes confidentiality weights so heavily that, nominal files can only be accessed 
by the individuals they refer to, restraining any public access to them, and creating 
some degree of ambiguity and uncertainty about how to proceed. Given this situation, 
the AGAE has adopted the legally required security levels for protecting personal data 
(Resolution of May 25'*' 2009), and has installed all the technical means and measures 
available to it in order to prevent any loss, misuse, alteration, unauthorised access or 
theft of any personal data provided. When applicable, the agency shall not assign any 
personal data of users without their written consent as indicated in art. 6.2 and 11.2 of 
the Protection of Personal Data Law. 

Simultaneously, AGAE staff and experts who participate in the agency’s quality 
assurance processes must work with confidentiality, and respect our internal quality 
system and the principles established in the Gode of Ethics (approved by the Board of 
Directors in July 2008) as indicated in art. 8 and 9 of AGAE’ Statutes. It is necessary to 
emphasise that AGAE further enforces a non-conflict of interest mechanism in the work 
of its external experts. The universities and research centres (for research evaluation) 


also have the right to state the existence of a conflict of interest in relation to the 
make-up of external assessment committees (panel experts). 

In line with the discussion presented above, when it comes to the publication of 
the decisions undertaken in the different programmes in place, the agency has to take 
into account a large number of different interests. The summary evaluation results of 
the institutional assessment programmes are published by the agency on its website, 
leaving to the evaluated institutions the role of making the detailed reports public. In 
the future, it may be desirable for the AGAE to publish the complete external reports, 
as some of the evaluated institutions are reluctant to publish the negative parts of 
their reports. Results from the accreditation of programmes containing personal 
information, such as the university staff programme, are directly provided to the 
interested person using legal pathways or a restricted area on the internet. Assessments 
of the rest of the programmes are published individually or in aggregate in the annual 
report. 

6.3 Concluding remarks 

In summary, we can conclude that for the AGAE, publishing information regarding the 
quality of institutions and programmes is not an easy task and a lot of thought has been 
given to it. In a competitive European Higher Education Area adequate and reliable 
information about the quality of institutions and programmes is very much desirable, 
and the role that our agency must play in terms of publicity of the evaluation reports 
is crucial. However, the agency’s legal framework, and more importantly, the political 
culture can sometimes limit our functioning and scope. In relation to the latter, we 
have answered already a couple of Parliamentary Queries questioning the anonymous, 
internationally accepted, peer-review system strictly followed by AGAE in some 
activities, such as evaluation of research proposals. 

In spite of the issues raised above, we believe that the agency is well aware of its 
mission. AGAE’s main activity is external quality assurance, and a systematic approach 
is applied in its working methods to achieve such an aim, with a special interest given 
to publicity and transparency. A general criterion for good quality in AGAE’S work 
is that the work shall be in accordance with stated goals of its Service Letter, under 
which the work is carried out with openness and clear communication externally, so 
that external parties can benefit from process and result. Lastly, the recently approved 
Quality Plan (June 2008) is a public promise that involves and binds the management 
of the agency, and demonstrates to users and providers once more, that AGAE is serious 
in wishing to improve its own performance. 

The balance is positive so far. In a Survey undertaken in 2008, overall users 
and providers consider that the service offered by AGAE is effective, transparent, 
consistent, systematic, fair and very supportive, with 68.2% of respondents reporting 
themselves as overall very highly satisfied. Optimising the balance between publicity 
and confidentiality should help us achieve a greater degree of acceptance as a quality 
assurance provider. 
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7.1 Introduction 

The French Agency for Evaluation of Research and Higher Education (AERES) was 
officially established in 2006, but started full operation only in March 2007. The 
number and volume of the structures and diplomas submitted to evaluation, and the 
size of the agency at that time required all its forces being fully engaged in this task. 

The methods used by the sections of the agency in this first period were close to those 
used by the CNE (comite national d’evaluation) and MSTP (mission scientifique, 
technique et pedagogique). The experiences gathered during the first year gave impetus 
for developing a more effective approach of the evaluations conducted by the agency. 
Two main axes emerged, as potentially useful for the agency, from: 

• organising an internal quality assurance (IQA) system covering all the agency’s 
activities; 

• integrating the activities of the three sections of the agency. 

This paper deals with the actions developed in the agency from March 2008 in order 
to implement an IQA system based on different models, among which the Standards 
and Guidelines for Quality Assurance in the European Higher Education Area (ESG)^. 
However, before entering into some details concerning the IQA of the agency, it is 
advisable to describe its status, missions, organisation and activities in a few words. 

7.2 Status, missions, organisation, and activities of the agency 

7.2.1 STATUS 

The agency was created in 2006 in accordance with the French law^“, and the principles 
of its organisation were described in a decree^k According to the will of the legislator, 
the agency is an “Independent Administrative Authority”. 

The most important characteristics shared by French independent administrative 
authorities are, briefly, that: 

• They are independent from the government and its administration which can 
neither order or command nor advise them, and their members are not removable; 

9 http://www.enqa.eu/pubs_esg.lasso 

10 http://www.legifrance.gouv.fr/affichTexte.do;jsessionid=4B5752F0642F4EAC1031EFB8EAC85BC2.tpdjol2v_3?cidTexte=JORFT 
EXT000000426953&categorieLien=id , Loi de programme pour la recherche n°2006'450 du 18 avril 2006 relative a la creation 
de I’AERES 

11 Decret n°2006-1334 du 3 novembre 2006 relatif a I’organisation et au fonctionnement de lAERES, version consolidee au 9 fevrier 
2009 


• They act on behalf of the State on the basis of a delegation of competences; 

• Their activity may result in recommendations and decisions/^ 

7.2.2 MISSIONS 

The missions conferred to the agency are intended to be a development of the former 
missions of: 

• CNE ( Comite national d ’evaluation) 

• CNER (Comite national d ’evaluation de la recherche) 

• MSTP (Mission scientifique technique et pedagogique) 

• DGES (Direction Generale de Tenseignement superieur) 

• GoNRS (Gomite national de la recherche Scientifique) 

In addition, the agency is responsible for: 

• approval of staff evaluation procedures 

• evaluation of all public research bodies (GNRS, INSERM, GEA) 

The agency aims to: 

• provide a reference for the evaluation of higher education institutions, diploma 
and basic/applied research; 

• encourage the development of national higher education institutions (HEIs); 

• provide a source of assistance to HEIs and relevant organisms in developing their 
own quality assurance culture; 

• fulfill the expectations of its stakeholders, national authorities, HEIs, and 
employers about the processes and outcomes of higher education. 

7.2.3 ORGANISATION 

• President: presides over, and is a member of the Board 

• Board of Agency : 25 members 

- 8 qualified personalities 

- 14 appointed members 

- 2 members of the Parliament 

• 3 sections headed by 3 Directors 

- Section 1: in charge of the evaluation of HEIs and Research bodies - Also 
ratifies procedures for staff evaluation 

- Section 2: in charge of the evaluation of the activity of Research Units 

- Section 3:in charge of the evaluation of courses and degrees of HEIs 

• 3 Advisers to the president 

• 1 General Secretary 

• Permanent staff (64 jobs): administrative and research staff 

• Over 80 part-time Scientific Advisers. 

7.2.4 EVALUATIONS CONDUCTED BY THE AGENCY IN 2008 

• Higher Education Institutions*: 


12 http://www.vie'publique.fr/decouverte-institutions/institutions/administration/organisation/etat/aai/qU'est'Ce-qU'autorite-- 
administrative-independante-aai.html 


- 33 HEIs: 18 French public universities, a Lebanese private university, 3 research 
bodies (among them : INSERM), 5 « Ecoles Franpaises al’etranger » (EFE), 

« College de France » , « Museum National d’Histoire Naturelle » , « Institut 
d’Etudes Politiques de Paris ». 

• Diplomas and formations: 

- Licenses: 832 

- Masters : 947 

- Doctoral schools* : 69 

• Research units* 

- 666 (maths-physics, chemistry:181; life sciences: 232; human sciences: 253). 

7.3 Internal quality management of the agency 

7.3.1 INTRODUCTION 

As stated in the Standards and Guidelines for Quality Assurance in the European 
Higher Education Area (ESG), agencies should have a published policy for the quality 
assurance of the agency itself, made available on its website. Accordingly, the agency 
has published some elements of its policy for quality management in April 2008. This 
constituted a public announcement that involved the whole agency. 

The recent creation of the agency, the large domain of its competences resulting 
in the evaluation of a high number of institutions, research bodies, diplomas and 
research laboratories were not facilitating factors. Another technical difficulty that the 
agency had to face was that this process was undertaken at a moment when evaluation 
methods had already been designed, and had thus to be incorporated in the IQA system 
in a way acceptable by the already overworked staff of the agency. 

The agency developed a global IQA system based on a process approach intended to 
describe properly all the activities of the agency, and to include all the evaluation grids 
in a logical framework, allowing their analysis and continuous improvement. 

The quality management system took into account the policy of the agency in terms 
of: 

• Independence resulting in a clear distinction between the evaluation processes 
(the agency) and the accreditation or financial decision processes (ministry). 

• Transparency: The choice of AERES was to publish the procedures and 
evaluation reports integrally. 

• Equity of the evaluations: The agency decided that, for the first time in France, 
all the research laboratories, within any university or research body, should 

be evaluated by expert teams using an on-site visit, and the specificity of the 
structures or domain should be taken into account in the choice of the experts 
involved. 

• Peer reviews: It is the choice and will of the agency that the evaluation reports 
result exclusively from peer reviews through over 5000 experts of the scientific 
community and other stakeholders (employers, students). Professional particulars 
of its members and experts are made public. 

• On-site visits of the agency’s expert panels are organised for the evaluation of 
HEIs, research units and doctoral schools. 

• No-conflict of interest mechanisms: The goal was to design a procedure which 
allows to avoid an overly indulgent or severe evaluation by the reviewers. 


7.3.2 INTERNAL QUALITY MANAGEMENT 

The agency develops an active internal quality management policy and, to this effect, 
has defined its specific goals (position statement) and created the adequate internal 
structures: “Comite de Pilotage” (Steering committee), “Cellule Qualite” (Quality 
management team) and “Commission Qualite” (Quality committee). The agency has 
defined the strategies and the schedule of the whole process according to its goals, 
and has made sure that proper human and financial resources were granted to quality 
management. 

7.3.3 DEEINING A STRATEGY EOR QUALITY MANAGEMENT 

The strategy for quality management of the agency relies on: 

• The necessary respect and fulfillment of the missions conferred to the agency by 
the law; 

• The choice made by the agency to adopt the ESG; 

• The use of relevant complementary quality standards (CAP 2006, ISO 17021, NF 
X 50 110). 

The approach is based upon: 

• The process development; 

• The internal evaluation of - and by - the agency, which implies: 

- The construction of specific standards adapted to the agency; 

- The training of the members of the agency. 

7.3.4 PROCESS APPROACH 

The first step was to identify the different processes involved in the global functioning 
of the agency. The process approach is summarised in a map reflecting the goals and 
activities of the agency. 

The processes belong to 4 categories: operational (skill) processes, support processes, 
management processes, and quality management processes. The first phase led to the 
implementation of all identified processes. The second phase, involving the specific 
training of the process teams, has been carried out. The dysfunction analysis and 
process improvement of operational and support processes is underway Concerning the 
management processes, the first phase is underway for the 7 identified processes. The 
quality management processes, which are expected to conclude the implementation of 
the approach, have still to be carried out. 

This important step has involved the participation of all staff and personnel of the 
agency. The reluctance observed sometimes at the beginning of the approach was not 
totally unexpected, and has progressively faded away when it became clear that the 
advantages of the method would by far exceed its constraints. 

There was a period when it was necessary to stimulate the “process teams”. 

Currently, the agency’s three sections have adopted - to variable extents - the approach 
which is now considered as central to all the evaluations conducted in and by the 
agency. However, it is also the experience of the agency that the processes and sub- 
processes have to be coordinated and controlled, and that the coordination is necessary, 
not to say crucial, to avoid any dysfunctions. 


7.3.5 THE CONSTRUCTION OF SPECIFIC STANDARDS 

In order to facilitate the constitution of its evaluation grids and its internal evaluation, 
the direction of the agency decided to elaborate specific standards, taking into account 
the missions of the agency, the Standards and Guidelines for Quality Assurance in the 
European Higher Education Area (ESG), and useful national or international standards 
for quality management (CAP 2006, ISO 17021, NF X 50 110 etc.). The standards were 
also to take into account the articulation of the activities between the three sections 
of the agency, considered as a first step to the integrated evaluation that the agency is 
planning to implement in the very near future. 

The question of the integrated evaluation of all evaluation activities has been the 
matter of considerable debate within the agency. Legal texts do not oblige the agency 
to integrate the evaluation carried out in the three sections, but, in fact, it is highly 
advisable that the evaluations of the diploma, courses and degrees as well as the 
evaluations of the research units/laboratories be available for a pertinent evaluation of 
the global strategy of the HEIs. However, this integration of the activities requires that 
a process be defined in common by the direction of the agency, and particularly by the 
three directors and their teams, and that its various steps be mastered and included in 
a convenient schedule. As a first step towards the integration of all evaluation activities 
of the agency, a joint evaluation of formations, research laboratories and HEIs was 
undertaken in 2008-2009. 

Three standards based upon the Standards and Guidelines for Quality Assurance in 
the European Higher Education Area (ESGl, ESG2 and ESG3) have been elaborated. The 
first version of the standards has been finalised in November 2008, and adopted by 
the AERES board in December 2008. The standards are used for the agency’s internal 
evaluation (self-assessment), which begun in June 2009. 

A specific standard for research is being prepared using ESG and standards 
established by INSERM, INRA and GNRS. It is under discussion within the agency 
staff. 

The text on specific standards was submitted to discussion within the agency, and 
presented to the board of AERES. However, even after its adoption by the board, and 
after being presented to the agency’s personnel at numerous meetings for further 
discussion, the text was still subjected to a strong questioning by the members of 
the agency. Discussion arose from people who considered themselves inadequately 
associated to the process, or who were not fully convinced of its necessity. As a 
consequence, the direction of the AERES re-opened the debate so that the standards 
would finally reach a consensus, which is considered an absolute requirement. 

7.3.6 ANALYSIS OF THE TEXTS TRANSMITTED TO HEIS 

Using the new tools resulting from its internal IQA, the agency has proceeded to the 
internal assessment of adequacy and conformity of the texts transmitted to HEIs in 
2009. Their conformity with the law, AERES mission statement, deontological rules, 
AERES Evaluation chart, AERES standards, and ESG was checked. The texts were 
improved, discussed with stakeholders, and published on the agency’s website. 

7.4 Concluding remarks 

Developing a quality management system is a strategic decision and, as such, it 
must have the support of the management of the institution, and the adhesion of its 


members. The process approach is the backbone of the system. After 18 months, the 
IQA system is operative and allows the agency to include its activities in a reasoned 
approach. The agency is currently preparing its external evaluation which is to take 
place in April 2010, almost exactly 3 years after the agency started its activities .The 
aim of the agency is, of course, to fulfill the requirements for membership in the 
European association for quality assurance in Higher Education (ENQA) and to be 
included in the European Quality Assurance Register (EQAR). To this effect, the 
agency has defined the tools it will use for its self-evaluation: coordination by a self- 
evaluation committee operating on a weekly basis; preparation of a self-evaluation 
guide; creation of working groups which analyse the compliance of all criteria with the 
ESG, collect all information available for each ESG reference, analyse the strengths and 
weaknesses, and propose improvements to the self-evaluation committee. 

The improvements resulting from this process should help to fulfill the expectations 
of the agency’s stakeholders: national authorities, higher education institutions, 
students and employers. 


Chapter 8: 

Quality as a fundamental index in 
the development of modern higher 
education 

Ivan Babyn and Viktoriya Lykova, Bologna Follow-up Group Representatives, Ukraine 


8.1 Introduction 

Quality in higher education is not only a national concern, but has become an 
international issue through academic, political, and commercial developments 
associated with globalisation, such as the rise of market forces in tertiary education 
and the emergence of a global market for skilled professionals and graduates. In some 
countries, the traditional providers of higher education are facing competition from 
transnational education providers, as well as from the emergence of local commercial 
providers. Through the internationalisation of higher education national systems, 
qualifications and individual higher education institutions have become exposed 
to the wider world. This exposure has stimulated a demand for better information 
and transparency about quality and standards in order to attract and retain students 
and staff, both national and international students, and to secure the recognition 
of qualifications. Quality assurance is a central thrust in the process of change 
in European higher education area (EHEA) following the signing of the Bologna 
Declaration. 

8.2 Student-centred learning 

Overcoming social and economic problems that higher education is confronted with, 
will depend on whether the potentialities and abilities of every individual as a subject 
of activity can be raised and realised to a maximum degree. In this context, the human 
resource becomes crucial in the struggle for the quality of the product of higher 
education as a sector of a country economy. The human capital is defined as a set of an 
individual’s characteristics based on such factors as health, natural abilities, education, 
professionalism and mobility. Every society member must be optimally suitable for 
professional activity at the expense of the maximal use of individual abilities.To 
achieve this aim, it is necessary to provide investment in higher education and upgrade 
training, and to accelerate progress in the fundamental task of modernisation of the 
national education system so that an individual could gain new knowledge, skills and 
competence needed in life-long education. 

8.3 National quality assurance 

In the rapidly changing environment of higher education, the maintenance of high 
quality and standards in education has become a major concern for higher education 
institutions and governments; thus, the demand for explicit quality evaluation and 
assurance processes has increased. The result has been the introduction of national 


quality assurance systems into many countries and the planned introduction of such 
systems into other countries. 

The challenges facing higher education worldwide include the following: 

• The need to assure quality and standards against a background of substantially 
increased participation - a process often referred to as the massification of higher 
education. This process accelerated throughout the latter part of the 20'*“ century 
as many countries began to consider that their economic and social future was 
dependent, in part, on the availability of quality higher education for the majority 
of the population rather than a small elite. However, expansion has not always 
been well planned or controlled; 

• The expansion in student numbers with either constant or declining (public) 
funding resulting in a lower unit of resource per student. This position has 
been compounded by the inefficient use of available resources. Examples of 
inefficiencies include overly high staff-student ratios, programme duplication in 
many small institutions/units with high unit costs, and under-utilised facilities; 

• Increased demand for accountability in higher education institutions as a result 
of deregulation and the granting of increased autonomy with regard to such 
matters as curriculum design, the selection of students, and the appointment 
of staff. However, increased autonomy has not always been accompanied either 
by financial authority, or by improved institutional management and strategic 
planning capabilities; 

• The meeting of new expectations in terms of the “employability” of graduates in 
the knowledge society; 

• The addressing of demands from a variety of stakeholders for increased and 
improved information about programmes and institutions and about the skills, 
competencies, and aptitudes which graduates possess; 

• The contribution to the achievement of social and political agendas such as access, 
inclusion, and equity. 

In addition to these factors, recent developments include the appearance of new 
providers of tertiary education, sometimes in competition with traditional public higher 
education, and new modes of provision, such as on-line learning, resulting from the 
information and communication technology revolution. An example of this type of 
competition is reflected in the “new technologies”. 

8.4 Conclusion 

The changing environment for higher education is a basic driver for the emergence and 
evolution of national quality assurance systems. To improve the quality of education 
service, the system of life-long education has been set up. This system promotes 
optimisation of the professional and qualification structure of the labor force and 
makes for the rise of its competitiveness level. 


Conclusion 


The internal quality assurance (IQA) of quality assurance agencies is constantly 
evolving. From recent experience we learn that adequate and functioning internal 
quality assurance procedures have a substantial impact on the successful achievement 
of the purposes of European quality assurance agencies 

In light of the articles presented in this publication, the key factor for efficient 
achievement of strategic purposes of internal quality assurance is transparency. 

Transparency involves accountability. Through transparency and accountability, 
stakeholders receive a truthful account of the agencies’ performance and can be more 
easily involved in the improvement of strategies for better performance. Ultimately, 
these improvements result in the building of trust between stakeholders and the 
agency, something that will benefit all involved parties. 

As quality assurance agencies strive to meet the constantly changing demands of 
their stakeholders, external evaluations for the mapping of their overall performance 
have proven to be an important tool. However, the articles in this publication 
demonstrate that internal evaluation processes, which are less visible from the 
outsider’s perspective, are equally important in building trust. Particularly beneficial 
are self-assessment procedures that involve agency staff and that include continuous 
internal reflection, feedback surveys and open self-critique. 

Quality assurance agencies are bound to publish to their stakeholders and providers 
not only the criteria and procedures used, but also precise information about all their 
functions and decisions undertaken in different procedures. Publishing complete 
external reports, including the negative parts of them, does not necessarily occur in 
every agency. However, agencies may seek to publish complete reports. This way, the 
agency can enhance its openness toward its stakeholders. At the same time, however, 
the agency must also strike a balance between public transparency and confidentiality 
for the protection of sensitive material and personal data. 
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13:00 Welcome and introduction to the seminar 

Tibor Szanto, Vice-President, ENQA 
Karl Dittrich, Chairman, NVAO 

Henri Ponds, Policy Advisor, NVAO, IQA Steering group representative 

13:30 Setting the scene: Internal quality assurance - common challenges among 
ENQA agencies 

Guy Aelterman, ENQA Board, Review Committee 
Riitta Pyykko, Professor, Chairman, EINHEEC 

Discussion 

14:30 Coffee break 

15:00 How can an agency better measure that its processes and results reflect 
the mission and goals of quality assurance (as stated in the ESG) 

Matti Kajaste, Adviser, EINHEEC 

Douglas Blackstock, Director of Administration and Company Secretary, QAA 


15:45 Breakout sessions on ‘How can an agency better measure that its 

processes and results reflect the mission and goals of quality assurance 
(as stated in the ESG)? 

Introduced by Valerie Liechti, Scientific Collaborator, OAQ 
Topics: 

• How can an agency better measure that its processes and results reflect the 
mission and goals of quality assurance (as stated in the ESG)? » two groups as 
a follow-up to the presentations by Matti Kajaste and Douglas Blackstock 

• The use of internal mechanisms as means to ensure the fulfillment of mission 
and goals. 

• The use of external feedback as information on fulfillment of mission and 
goals. 

16:30 Two case presentations on ‘How do agencies follow recommendations 
from the review team and the review committee?’ 

Luna Lee Solheim, Senior Adviser, NOKUT 

Tibor Szanto, Secretary General, Hungarian Accreditation Committee 

-17:30 

19:00 Social programme 

20:00 Dinner 


22:00 Transport to hotels 


Day 2 

16 June, 2009 


08:45 

09:00 

09:30 

11:00 

12:30 

13:30 

14:30 

15:00 


Introduction of the ENQA IQA Steering group 

Internal procedures in agencies: How do we guarantee efficient and 
excellent operations in our agencies? 

IQA Steering group representatives Nuria Comet Serial, Responsible for the 
Internal Quality Assurance System, AQU Catalunya and Henri Ponds, 

Policy Advisor, NVAO 

Breakout sessions on ‘Internal procedures in agencies: 

How do we guarantee efficient and excellent operations in our agencies?’ 
Topics: 

• Composition of panels 

• Training of experts 

• Consistency in decision making 

• Communication and the impact of accreditation decisions 
Coffee break during the breakout sessions 

Global trends on IQA 

Crossroads - Axel Aerden, International Policy Advisor, NVAO 
Nadia Badrawi, President, ANQAHE 

Lunch 

An outsider’s view on IQA 

Lee Harvey, Visiting Professor, Copenhagen Business School 

Concluding session 

Signe Ploug Hansen, Head of unit, EVA, IQA Steering group representative 

End of seminar 
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The Accreditation Organisation of the Netherlands and Flanders (NVAO) 
hosted the annual meeting of the ENQA IQA Group in The Hague in June 2009. 

This seminar was a successful follow-up to the first ENQA IQA seminar in Barcelona 
in 2008 on Internal quality assurance systems and the ESG. The seminar focused on 
the theme Internal Quality Assurance - Facing Common Challenges. Staff members 
of ENQA member agencies discussed the common, tangible challenges they 
frequently encounter in their daily operations. 



